Method for optimal body or face protection with adaptive dewarping based on context segmentation layers

ABSTRACT

A method for enhancing a wide angle image to improve the perspectives and the visual appeal thereof wide-angle images uses custom adaptive dewarping. The method is based on the scene image content of recognized objects in the image, the position of these objects in the image, the depth of these objects in the scene with respect to other objects and the general context of the image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 16/898,501, filed on Jun. 11, 2020, entitled“Method for Optimal Body or Face Protection with Adaptive DewarpingBased on Context Segmentation Layers,” currently pending, which claimsthe benefit of U.S. Provisional Patent Application No. 62/859,861, filedon Jun. 11, 2019, entitled “Method for Adaptive Dewarping Based onContext Segmentation Layers,” now expired, the entire contents of all ofwhich are incorporated by reference herein.

BACKGROUND OF THE INVENTION

Embodiments of the present invention relate to the field of photographyand more specifically on how image distortion in wide-angle images canbe corrected differently depending on the image context, thesegmentation layers and/or the depth of objects visible in the image.

In photography, with narrow-angle lenses having full field of view under60°, it is generally desirable to have images in which the straightlines in the objects remain straight. This is achieved by having imagesfollowing as closely as possible the rectilinear H=f*tan(0) relationbetween the image height H and the field angle θ, which remain doable innarrow-angle lenses. With a quite limited full field of view below 60°,this type rectilinear H=f*tan(0) relation do not impact significantlythe object proportion on periphery of the image. The images followingexactly this relation are said to be without optical distortion. Foroptical lenses that do not follow exactly this relation for all fieldangle θ, the resulting image from these lenses are said to have someoptical distortion. This optical distortion is especially present inwide-angle images having full field of view over 60°. Correcting theresidual image distortion of wide-angle image or modifying it on purposeare known techniques in image processing that are often used when theoptical lens itself cannot be designed to create the desired projectionfor the desired application.

While the rectilinear projection is ideal to keep the straight lines ofthe object straight in the image, it is sometime not the projectioncreating the most visually pleasing images in photography. One suchexample is the group selfie, or groupie, with wide-angle lenses in whichpeople are located at various positions of the field of view. The peoplein the center appears with normal proportions, but the people toward theedges appear stretched and deformed because of the rapidly increasingnumber of pixels/degree of this projection. This unpleasant visualeffect on human faces is not only visible in lenses with a rectilinearprojection, but in every lens not specifically designed to keep theproportions visually pleasing.

Some image processing algorithm or some lenses are designed specially tolimit this undesirable effect toward the edges by limiting the rapidlyincreasing number of pixels/degree toward the edge at the expense ofcreating curved lines. In other words, even with perfect calibration ofthe lens and the dewarping algorithm, the dewarping algorithm will onlybe able to correct either straight lines or the faces proportions sincethese two corrections require different dewarping projections. If thecorrection algorithm is optimized to offer more visually pleasing imagesof human located toward the edges of wide-angle images, a process calledbody and face protection, they will have the undesirable consequence ofdoing so by adding geometrical distortion in the images and theresulting images will have curved lines even if the original objectscene consist of straight lines. On the opposite, if the correctionalgorithm is optimized to straighten the lines in the image, it willworsen the proportions of human located toward the edges.

For example, the image distortion transformation method presented inU.S. Pat. No. 10,204,398 B2 is used to transform images the distortionfrom an original image from an imager to a transformed image in whichthe distortion of the image is modified according to a pre-configured ora selected target image distortion profile. Even if this targetdistortion profile can be asymmetric, for example to maintain a similarfield of view, this target distortion profile is however applied to thewhole image without consideration for the position of objects in theimage or their depth. As such, when using this method, one can improvethe appearance of straight lines while worsening the appearance ofpeople in the image or the opposite.

Some other existing warped image geometries corrections method alreadyexists, like perspective tilt correction with the horizon as presentedin U.S. Pat. No. 10,356,316 B2. However, these methods can only correctthe perspective for the whole image and cannot apply the correction to aspecific element.

Another issue to overcome is the fact that real lenses from a massproduction batch all differ slightly from each other due to tolerancingerror on the shape, position or orientation of each optical elements.These tolerancing error create slightly different distortion profile foreach lens of a mass production batch and so residual geometricaldistortion can be present in the images even after dewarping the imagebased on the theoretical distortion curve for this mass-produced lens.

In order to have body and face protection, that is having the mostvisually appealing human proportions, while still making the straightlines in the object appear as straight lines in the images, some moreadvanced image processing algorithms exist, applying a specific imagedewarping depending on the content of the image. However, when applyinga correction for a foreground object or person, these algorithms havethe undesirable consequence of breaking the perspectives in thebackground. A new method is required to overcome all these issues.

BRIEF SUMMARY OF THE INVENTION

To overcome all the previously mentioned issues, embodiments of thepresent invention present a method of adaptive dewarping to applydifferent dewarping algorithm based on scene context, position of theobject and/or depth of objects present in the image. From an initialinput image, the method first applies an image segmentation process toseparate the various objects visible in the original image. Then, thedepth of each object is used to order all the objects by layersaccording to their object type and their depth with respect to thecamera. The depth value used to separate the layers is either anabsolute depth measurement or a relative depth between the objects andcan be calculated using an artificial intelligence neural networktrained to infer the depth from 2D images, from parallax calculationusing a pair of stereo images, from a depth measurement specific devicelike a time of flight sensor, structured light systems, lidar, radar, 3Dcapture sensor or the likes. Each category of object that is recognizedin the original image will be dewarped using a specific dewarpingalgorithm or projection. For example, if human faces are detected in theoriginal images, a specific dewarping algorithm to avoid stretching thehuman faces and make them more visually appealing will be used on thehumans, a process called face protection. If the same original imagealso contains buildings, the adaptive dewarping algorithm will apply adifferent dewarping on the building to keep the lines straight. There isno limit according to the present invention to the types of objects thatcan be recognized by the adaptive dewarping method and to the dewarpingto be applied on them. The dewarping to apply can be defined in advanceas a preset for a specific object type, e.g human face, building, etc.or can be calculated to respect object well-known characteristics suchas face proportions, human body proportions, etc. This method is appliedon each segmented layer and depth layer, including for the backgroundlayer. The background layer consists of objects far from the camera andnot having a preset distortion dewarping. In preferred embodimentsaccording to the present invention, the background is dewarped in orderto keep the perspective of the scene undistorted. Compared to existingprior art, the adaptive method allows to apply a different dewarping toeach kind of object based on type, layer, depth, size, texture, etc.

In some cases, when deforming a given layer, the adaptive dewarping cancreate a region of missing information in a layer behind because thelayer behind is applied a different dewarping. Only when this happen, anadditional image completion step can be applied on the resulting imageto further make it more visually appealing. This image completion stepconsists of separating the objects after adaptive dewarping based oncontext in several depth layers. Some depth layers have missinginformation and some other depth layer are without any missinginformation. A completion algorithm is then used on the layers withmissing information in order to fill the region with missinginformation. This can be done by applying a blur based on the textureand color surrounding the missing information zone, applying a gradientline that gradually change the color from the color on one side to thecolor on the other side of the missing information region, using anartificial intelligence network trained to complete the missinginformation of pictures or the likes. The completion algorithm outputcompleted depth layers that can then be merged back in a single imagewith filled information in which the perspective corrections wereapplied, the people shapes were corrected avoid unpleasant stretchingand the missing background information was filled with the completionalgorithm.

In some embodiments according to the present invention, the dewarpingprojection for the background layer depend on the context identified inthe foreground objects.

In some embodiments according to the present invention, the adaptivedewarping method is used to either maximize the straightness of thelines in the images compared to the original lines in the object scene,maximize the output image full field of view compared to the originalimage full field of view and/or maximize the conservation of theproportions in the output image compared to the real proportions in theobject scene.

In some embodiments according to the present invention, instead ofcompleting the missing information with a completion algorithm, therelative magnification of front layers is increased to cover the regionsof missing information in layers behind. In certain case, this techniquemight make front objects appear bigger or closer than they were used tobe in the original image.

In some embodiments according to the present invention, the selection ofthe dewarping projection for a background layer depend on the detectedcontext of an original wide-angle image.

In some embodiments according to the present invention, the processingincludes creating a virtual camera centered on the element having thewarped geometry, applying a rectilinear correction on the virtual cameraand translating the result to a correct location in the final image.

In some embodiments according to the present invention, the method foradaptive dewarping based on context and segmentation layers includeprocessing done by a processor inside a physical device also creatingthe original wide-angle image with an imager and displaying the finalimage to a display screen.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The foregoing summary, as well as the following detailed description ofa preferred embodiment of the invention, will be better understood whenread in conjunction with the appended drawings. For the purpose ofillustration, there is shown in the drawings an embodiment which ispresently preferred. It should be understood, however, that theinvention is not limited to the precise arrangements andinstrumentalities shown.

In the drawings:

FIG. 1 shows the resolution curve of a rectilinear image;

FIG. 2 shows how existing wide-angle cameras with no or small departurefrom rectilinear projection create visually unpleasing views;

FIG. 3 shows the resolution curve of an image from a wide-angle camerawith departure from rectilinear projection;

FIG. 4 shows how existing wide-angle cameras with large departure fromrectilinear projection also create visually unpleasing views;

FIG. 5 shows a basic method to apply image correction to make them morevisually pleasing, but while affecting the perspective;

FIG. 6 shows the method for adaptive dewarping in its previous form;

FIG. 7 shows the method for adaptive dewarping based on contextsegmentation and depth layers;

FIG. 8 shows the method for filling missing image information after theadaptive dewarping method has been applied;

FIG. 9 shows the method for hiding missing image information after theadaptive dewarping method has been applied;

FIG. 10 shows the method in which the dewarping projection of thebackground layer depends on the context of the objects in theforeground;

FIG. 11 shows the steps of the algorithm according to the method forcontext-based adaptive dewarping based on depth and segmentation layerfor ideal of face protection; and

FIG. 12 shows an example embodiment of a physical device capturing theoriginal wide-angle image, processing it and outputting the final imageon a display screen.

DETAILED DESCRIPTION OF THE INVENTION

Certain terminology is used in the following description for convenienceonly and is not limiting. The words “right”, “left”, “bottom”, and “top”designate directions in the drawings to which reference is made. Theterminology includes the above-listed words, derivatives thereof, andwords of similar import. Additionally, the words “a” and “an”, as usedin the claims and in the corresponding portions of the specification,mean “at least one.”

It should also be understood that the terms “about,” “approximately,”“generally,” “substantially” and like terms, used herein when referringto a dimension or characteristic of a component, indicate that thedescribed dimension/characteristic is not a strict boundary or parameterand does not exclude minor variations therefrom that are functionallysimilar. At a minimum, such references that include a numericalparameter would include variations that, using mathematical andindustrial principles accepted in the art (e.g., rounding, measurementor other systematic errors, manufacturing tolerances, etc.), would notvary the least significant digit.

FIG. 1 shows the theoretical resolution curves of perfect rectilinearimages respectively with 40° half field of view at 100 and with 70° halffield of view at 150, corresponding to a full field of view ofrespectively 80° and 140°. Most narrow angle imaging lens, having a fullfield of view under 60°, designed for use in photography applicationstarget to have as low as possible image distortion by following as muchas possible a rectilinear image projection. In a rectilinear lens, therelationship between the image height H on the image sensor and the halffield of view angle θ in the object plane follow as closely as possiblethe equation H=f*tan(0). This projection is usually followed bynarrow-angle lens having a full field of view under 60°. However,wide-angle lenses, also called panoramic lenses, having a full field ofview greater than 60°, do not usually follow this H=f*tan(0) equationexactly. An image that follow perfectly this H=f*tan(0) equation, eitherdirectly from capturing on an image sensor the image from the imaginglens or after any hardware or software distortion correction ordewarping, is said to have no image distortion. Any departure from thisequation is called image distortion, geometrical distortion or opticaldistortion and is usually avoided in photography. Departure from thisequation is also related to TV distortion, in which the corners of arectangular objects appear expended or compressed in the image insteadof perfectly rectangular. The graphs 100 and 150 show the resolutioncurves as a function of the half field of view. The resolution curve isobtained from the taking the mathematical derivative of the positioncurve as a function of the half field of view angle θ. On the graph 100for the case of a full field of view of 80°, the value 110 representsthe magnification of 1x in the center of the field of view, at a halffield of view angle θ of 0°. Alternatively, instead of calculating theresolution as a ratio of magnification with respect to the center, itcould also be calculated in mm/degree, in mm/radian, in pixels/degree orin pixels/radian, or the likes. The value of pixels/degree is especiallyuseful when the image sensor consists of pixels of constant dimension asit is most often the case. For a half field of view 0 value of 40°, theresolution value 112 is 1.7x larger than the resolution in the center110 for the theoretical rectilinear projection and the resulting imagesalready appear stretched. On the graph 150 for the case of a full fieldof view of 140°, the value 160 represents the magnification of 1x in thecenter of the field of view, at a half field of view angle θ of 0°. Fora half field of view 0 value of 45°, the resolution value 162 is 2xlarger than the resolution in the center 160 for the theoreticalrectilinear projection and the resulting images appear even morestretched. For wider half field of view angle θ, the difference inresolution from the center to the edge become increasingly bigger andthe image become even more stretched and unpleasant for somephotographic applications. For example, with the theoretical rectilinearprojection, at a half field of view value of 60°, the resolution 164 is4 times bigger than the resolution 160. At a half field of view value of70°, the resolution 166 is 8.55 times bigger than the resolution 160. Atlarger half field of view value, the resolution keeps increasing untilreaching infinity at a half field of view angle of 90°.

FIG. 2 shows example images of a group selfie, or groupie, as it wouldappear either when captured by a theoretically perfect rectilinear lensor after hardware or software correction of the image distortion to getan image with a perfect rectilinear projection. In example image 200,the full field of view in the diagonal direction of the image is 80°while in example image 250, the full field of view in the diagonaldirection of the image is 140°. In the example image 200 with 80°diagonal field of view, we can see that the person with its head 212 inthe center of the image appears normal because in the central area ofthe image, the resolution is almost constant. However, for the person214 having its head toward the edges, the face is stretched in adirection away from the center and it looks deformed. This phenomenon ofrectilinear projection is visually not pleasing for consumer photographyapplications, but this stretching is required to keep the lines in theobject scene to appear straight as in horizonal line 220, vertical line230 or vanishing line 240. Similarly, in the example image 250 with 140°diagonal field of view, we can see that the person with its head 262 inthe center of the image appears normal because in the central area ofthe image, the resolution is almost constant. However, for the person264 having its head toward the edges, the face is stretched in adirection away from the center and it looks deformed. For the person 266having its head at a half field of view angle closer to the corner of70°, this stretching is even more visible. Again, this phenomenon ofrectilinear projection is visually not pleasing for consumer photographyapplications, but this stretching is required to keep the lines in theobject scene to appear straight as in horizonal line 270, vertical line280 or vanishing line 290.

FIG. 3 shows the resolution curve of a more pleasing wide-angle image,either obtained directly from a wide-angle lens having a drop ofresolution toward the edges after having a maximum value in a regionbetween the center and the edge or obtained after modifying on purposethe image distortion with hardware or software dewarping or correctionalgorithm to avoid the undesirable effect of FIG. 2 . This resolutioncurve 300 having compressed zones in the center and at the edge of thefield of view and an expended zone in an intermediate zone of the imagelocated between the center and the edges of the image is only an exampleresolution curve creating more visually appealing images, but otherresolution curves creating more visually appealing images exist. Thiskind of resolution curve 300 having a given value of resolution 310 inthe center, increasing smoothly up to a maximum 312 and then droppingback to an edge value 314 is typical of some wide-angle lenses orultra-wide-angle lenses creating an expended zone in an intermediatezone of the half field of view and a compressed zone at the edge to havean image visually pleasing. In this example, the maximum resolution isaround a half field of view angle θ of 45°, having a magnification valueof approximately 2x as was the case with the rectilinear curve of FIG. 1, but many similar resolution curves with different maximummagnification value and location could be used to create an image morevisually pleasing. Also, in this example, the average magnificationvalue is about 1.5x, which would also be the magnification of anequidistant H=f*0 projection creating an image of the same size for thesame total field of view. In this example, to create an image morevisually pleasing, the magnification 310 in the center is below thataverage value, the magnification 312 at the maximum magnification isabove this average value and the magnification 314 at the edge of thefield of view is below this average magnification value. However, insome other embodiments according to the present invention, themagnification 314 at the edge of the field of view could also be abovethe average magnification value.

FIG. 4 shows two example images 400 and 450 of a group selfie, orgroupie, as it would appear respectively when captured by a lens havinga resolution curve like the one of FIG. 3 and after image warpingcorrection to a proportion saving projection, also called body and faceprotection, in order to avoid or minimize the undesirable effect of FIG.2 . In the top image 400, the head of the person standing in the center422 still appears normal. The head of the people 424 and 426 standing atrespectively at half field of view angle of 45° and 70° also appearsmore normal than in FIG. 2 because the selected projection having theresolution curve of FIG. 3 does not have a large increase of resolutiontoward the edge as it was the case with the curve of FIG. 1 . Thisresult for the face is visually more pleasing in photography. However,because the lens does not follow the rectilinear projection mappingequation H=f*tan(0), there is geometrical distortion in the image andthe straight lines in the object scene do not appear straight in theimage, as seen with the curved horizontal line 430 and vertical line435. However, in this example, the vanishing lines 440 remain straightsince they are oriented in the radial direction from the center of theimage. In the bottom image 450, an additional image processing was doneto get an image with a perfect proportion saving projection, also calledface and body protection. With this projection, the head of the people472, 474 and 476, standing respectively in the center, at a half fieldof view angle of 45° and at a half field of view angle of 70°, all havesimilar proportions because of the body and face protection correction,which is also visually pleasing for a group selfie picture. However,because this face protection projection keeping the proportions does notfollow the rectilinear projection mapping equation H=f*tan(0), there isgeometrical distortion in the image and the straight lines in theobjects does not appears straight in the image, as seen with the curvedhorizontal line 480 and vanishing lines 490. In this example projection,the vertical lines 485 are kept straight in the images, but this is notalways the case.

FIG. 5 shows a simple method according to the present invention in orderto keep the original straight lines of the object straight in the imageas well as making sure that some objects like people faces are not toostretched when they are close to the edge of the image. This methodallows to enhance the original wide-angle image based on the imagecontext. In the original image 500 having a rectilinear projection,either from using a lens with H=f*tan(e) distribution function or aftercorrecting the distortion with image processing, the person in thecenter 522 appear normal and the people 524 and 526 toward the edgeappears increasingly stretched. With this original image, all thehorizontal, vertical and vanishing lines 530, 535 and 540 are straight.This original image 500 is like the case of FIG. 2 . One simple methodto improve the appearance of the image is to correct locally the shapearound stretched objects like the head 524 and 526 while keeping linestraight, resulting in the example image 550. The method starts byreceiving an original wide-angle image having at least one elementhaving a warped geometry. Here, a warped geometry can be of any kind,including uneven or stretched proportions, curved lines when theoriginal object is straight, unpleasant optical distortion or any otherunpleasant artefact visible in the original wide-angle image. The methodthen create at least one classified element by classifying from theoriginal wide-angle image the at least one element having the warpedgeometry. Here, the classification of the element can be based onvarious methods, including, but in no way limited to, based on a shapeof the at least one element, a position of the at least one element inthe original wide-angle image, a depth of the at least one elementcompared other elements in the original wide-angle image or the likes.The method then allows to create a final image by processing theoriginal wide-angle image to dewarp the warped geometries. This type ofprocessing with correction algorithm can be done by an A.I. algorithmusing deep learning to correct the shape of objects or a traditionalimage morphing algorithm that correct the unpleasing shape by knowingits position in the image and the resolution curve creating it. In thisfinal image, the correction is on the whole image withoutdifferentiating the front layer from the background. The correction canbe done by transforming either the texture mesh or the display mesh.Alternatively, the correction can also be done by dewarping the imagepixel by pixel. Corrections to the images make the people toward theedge 574 and 576 to appear more normal like the person standing in thecenter 572, that is body and face protection was applied. However,because of the local morphing being applied to the whole image, theundesired result from correcting the foreground is breaking theperspective in the background. In this example, the straight lines thatare not touching or hidden by a foreground object remain straight likethe lines 580 and 590. This is not the case of the straight lines hiddenbehind a foreground object like the people 574 or 576. The lines, whichare continuous in the real object scene and the resulting originalwide-angle image, are now discontinuous in the final image because ofthe correction that was applied on the foreground. This is especiallyvisible with segments of lines 581 and 582 forming a continuoushorizontal line in the object space but a discontinuous line in thecorrected image or with segments of lines 585 and 586 forming acontinuous vertical line in the object space but a discontinuous line inthe corrected image. This figure shows an example of the simplecorrection method based on classification and without segmented layers,but the method is not limited to people and could be applied to variousother objects to correct their warped geometries due to the non-linearmagnification in the image across the field of view.

FIG. 6 shows another example of the method for adaptive dewarping basedon image context in its simple form using element classification. Inthis example FIG. 6 , an original image 600 has several warpedgeometries, including curved lines 610, unequal proportions between thehuman face as seen by the difference in size between the faces 612 inthe center and 614 at the edge and the image has a diagonal field ofview of 140°. This original image 600 is only an example original image,either captured directly from a wide-angle imager or after someprocessing has been applied, but the method according to the presentinvention is not limited to this scene content or to any value ofdiagonal full field of view. Some of the existing methods correctedcompletely the proportions by using a proportion saving projection. Thisis like in the previous example of FIG. 4 , represented here by theimage 620 in which the lines 630 are even more curved, but the faceproportions are equal, as seen in the equal size for the face in thecenter 632 and the face at the edge 634. In this example, the proportionsaving projection could be, but not limited to, an equirectangularprojection, a cylindrical projection or any other custom projection.Some other of the existing methods straightened completely the lines inthe image as in the example of FIG. 2 , represented here by the image640 in which the lines 650 are straight, but the face proportions areeven worse than in the original image as seen by the larger differencein size between the face 652 in the center and the face 654 at the edgethan in the original image. In both of these existing methods,conserving the original full field of view was not possible whenmodifying the image projection. With the method of the presentinvention, the simple form of the adaptive dewarping method based onimage context is used to process the original wide-angle image toequally maximize the straightness of the lines in the final imagecompared to the original lines in the object scene, maximize the finalimage full field of view compared to the original wide-angle image fullfield of view and maximize the conservation of the proportions in thefinal image compared to the real proportions in the object scene. Again,the method starts by receiving an original wide-angle image withelements having warped geometries. The method then create at least oneclassified element by classifying from the original wide-angle image theat least one element having the warped geometry. Here, theclassification of the element can be based on various methods,including, but in no way limited to, based on a shape of the at leastone element, a position of the at least one element in the originalwide-angle image, a depth of the at least one element compared otherelements in the original wide-angle image or the likes. The method thenallows to create a final image by processing the original wide-angleimage to dewarp the warped geometries and maximize the field of view ofthe final image. The correction can be done by transforming either thetexture mesh or the display mesh. Alternatively, the correction can alsobe done by dewarping the image pixel by pixel. The resulting image 660has lines 670 that are straighter than in the original image 600, butless than in the image 640. The resulting image 660 also has faceproportions more equal than in the original image 600, but less than inthe image 620 as seen by comparing the ratio of the face in the center672 to the face at the edge 674. Finally, the diagonal full field ofview in the image 660 is kept as close as possible to the value of 140°from the original image 600 to avoid creating a zone with no informationeither in the corners or on the sides of the images or having to cropthe image to avoid such zone with no information. The level of idealbalance between the three items to maximize with this simple form of theadaptive dewarping method depend on which compromise is acceptable forthe desired application. In some embodiment, each of the three items tomaximize could be assigned an adjustable correction weight to adjust thelevel of processing done on the original wide-angle image. Theseadjustable correction weights are either pre-defined for example by therequirement of the application or selected by the user according to itspreference. Depending on the input original image content, the contextand the application, the level to which the straightening of the curvedlines, the conservation of the field of view and the conservation of theobject proportion which is applied can vary according to the simple formof the method for adaptive dewarping according to the present invention.In some embodiments, the processing step of the method could be done byan artificial intelligence algorithm.

FIG. 7 shows the preferred method according to the present invention foradaptive dewarping based on image context segmentation and segmentationlayers. The method receives as an input an original wide-angle image 700having a plurality of elements, each element being in one of aforeground or a background of the original wide-angle image, one or moreof the elements having a warped geometry. These warped geometries can beof any kind, including uneven or stretched proportions, curved lineswhen the original object is straight, unpleasant optical distortion orany other unpleasant artefact visible in the original wide-angle image.This wide-angle image can be of any field of view, but generally theunpleasant effects shown in FIG. 2 and FIG. 4 are mostly visible inwide-angle images with over 60° of full field of view. In a preferredembodiment according to the present invention, this original wide-angleimage 700 is directly captured by an imager having an optical systemcomprising at least a camera module and a wide-angle optical lens withor without departure from the rectilinear projection. This wide-anglelens generally has a diagonal field of view of at least 60°. In otherembodiments, this optical system consists of any combination ofrefractive lens element, reflective mirror element, diffractive element,meta-surface, or any other optical element helping to form an image inthe image plane of the camera. In some other embodiments according tothe present invention, the original image 700 has already been processedby a processor to either correct the original distortion from the cameramodule, improve the image quality or applying any other imageprocessing. Alternatively, the original wide-angle image 700 could becreated inside a imager with a processor from combining multiplenarrow-angle images or completely computer-generated. The originalwide-angle image 700 has elements that are visually unpleasant for ahuman observer. In this example figure, in no way limiting the scope ofthe present invention, the elements are a human 702 in the center thatappears normal, a human 703 at the edge with a face unpleasantlystretched, a tree 704 at the edge that is deformed, a building 706 onthe edge that even if it is straight in the object scene, it appearscurved due to image distortion, a building 707 which appears normallystraight because of its position in the center and a background 708consisting of various far away objects as mountains or the Sun. Afterreceiving the original wide-angle image, the method follows with theobject segmentation and depth analyzing step 710 based on element depthand image context. This first processing step to segment the originalwide-angle image into a plurality of segmented layers is done via asoftware algorithm, a hardware algorithm or an artificial intelligencealgorithm trained or not via a neural network. This first processingstep could be executed inside a processor, a CPU, a GPU, an ASIC, a FPGAor any other device configured to perform image segmentation orexecuting algorithms. In some embodiments, this processing step is doneinside the same physical device on which the imager with the wide-anglecamera module is located. In other embodiments, this processing step isdone inside a different device on which the adaptive dewarping isrequired to improve the image. The segmentation processing step analysesthe original wide-angle image content and separate its various elementsin various segmented layers, each segmented layer including at least oneof the elements. This segmentation can be done depending on the elementclassification and also optionally the depth analysis. This depthanalysis step separates the various segmented layers based on distanceof the various elements in the original wide-angle image. Thissegmentation step can also be based on the shape or the position in theimage of the various elements. The depth of the elements, especiallythose in the foreground of the original wide-angle image, can beestimated with a depth estimation algorithm, including A.I. neuralnetworks trained to deduce the relative depth of an element compared toother elements from a single image, an algorithm reconstructing the 3Dstructure of a scene from analyzing the difference between successiveframes of a video sequence when the camera is voluntary or involuntaryin motion combined with the gyroscope information from the device, orany other algorithm used to estimate, calculate or measure the depth ofan element in the scene. When the depth estimation is done by a neuralnetwork, the network can be of any shape, including, but in no waylimited to, neural network with convolution layers. The network can alsoconsist of sub-networks or sub layers, each doing separate tasks,including, but in no way limited to, convolution, pooling (Maximumpooling, average pooling or other type of pooling), striding, padding,down-sampling, up-sampling, multi-feature fusion, rectified linear unit,concatenate, fully connected layers, flatten layers or the likes. Inother embodiments of the current invention, the depth of each object canbe calculated from a stereo pair of image captured from differentpositions in order to calculate differences due to parallax, from atime-of-flight hardware module, from structured light systems, from aLIDAR system, from a RADAR system, from a 3D capture or by any othermeans to estimate, measure or calculate the distance of objects visiblein the image. With all the above examples of methods or systems toevaluate the depth, the resulting depth information can be eitherabsolute or relative. In the case of relative depth, the depth does nothave to be precise and it can be only an information that discriminatethe relative position of each layer. One such example of a relativedepth measurement, in no way limiting the possible methods to rank thedepth of the layers, is the relative depth measurement based onsuperposition. In the image 700, the head of the person 703 partiallyhides the tree 704 because of the superposition, allowing the depthestimation algorithm to rank the relative depth of the tree 704 as beingfarther away than the person 703 even if absolute distance are notavailable. In some embodiments of the current invention, both thesegmentation algorithm based on image context and element classificationand the depth analysis are executed together, and they help each otherto improve the results of their analysis. In the example of FIG. 7 , thesegmentation and depth analysis algorithm created five different layersbased on context and depth of the objects. The context analysis can bethe result from a classification algorithm performed at the same time asthe segmentation algorithm. This classification algorithm is used toclassify each segmented element in identified categories. In thisexample, the first and second layers 720 and 725 are for people. Eachlayer from the segmentation algorithm correspond to a predefined rangeof distance. For this reason, even if the two humans 702 and 703 fromthe original image 700 were the closest objects from the wide-anglecamera, their distance from each other was greater than thepredetermined minimum step and hence they form two different layers 720and 725. In the layer 720, person 722 still appears stretched andvisually unpleasing as did the person 703 in the original image 700. Inthe layer 725, the person 727 still appears correctly as person 702 fromthe original image 700. In this example, the third layer 730 comprisesof unrecognized objects by the segmentation algorithm or recognizedobjects for which no particular adaptive dewarping is required, like thetree 734. The fourth layer 740 is for buildings in which the buildings742 and 744 are still distorted as the buildings 706 and 707 in theoriginal image 700. Here, the two buildings 742 and 744 from the layer740 were considered at the same distance from the camera compared to thepredetermined minimum distance step. Since they are from the sameclassification type and at the same depth, the segmentation and depthanalysis algorithm 710 sorted them in the same layer 740. The tree 734was also considered at the same distance than the two buildings 742 and744, but since the segmentation algorithm found them to be from twodifferent kinds, they are in different layers 730 and 740. Finally, thelast layer 750 is the background, which consist of all objects like themountain 755 far away in the image that will not be affected by theperspective corrections. The method then process at least one of thesegmented layers to at least partially dewarp any of the one or moreelements having the warped geometry, creating a dewarped layer. This isdone by the adaptive dewarping 760 depending on the image context anddepth of the objects in the original wide-angle image. In a preferredembodiment, the specific dewarping process to be applied on thesegmented layer depend either on a position of the segmented layer inthe original wide-angle image or on a classification of the elements inthe segmented layers. The context of the original wide-angle imagedepends on these and is often determined automatically from analyzingthe elements from original wide-angle image 700. In other cases, thecontext could as well be determined with information from thesegmentation and depth from each layer obtained from the algorithm 710.Alternatively, the exact information and parameters of the adaptivedewarping to be applied could be transferred to the adaptive dewarpingalgorithm via metadata or a maker in the image, a user input, aselection from a list of adaptive dewarping algorithm or selectedautomatically according to the application. In this example, since theoriginal image had a segmentation layer with people, a custom dewarpingwith body and face protection based on the context 760 specifically forpeople will be applied on layers 720 and 725 to get respectively thedewarped layers 765 and 770. This custom dewarping for people does nottry to keep the perspective or straight lines of the objects, but ratherto keep the shape of human visually pleasing no matter where they are inthe field of view. Next, the custom dewarping for unknown orunrecognized objects is applied to layer 730 to get dewarped layer 775.This custom dewarping improve the shape of objects toward the edge ofthe image as if they were imaged in the center of the image based on thedifference of magnification from one edge to the other edge of theobject, but without specific correction as for known objects thatrequire a specific correction (building, people). Next, the adaptivedewarping is applied on the layer 740 to get the dewarped view of thebuilding 780. For buildings, it is important for the image to bevisually pleasant to keep straight lines and hence the projectionapplied on this layer keeps the lines straight. Finally, the backgroundlayer 750 can also optionally be dewarped if required to get the desiredprojection, obtaining the dewarped layer 785. The last step of themethod according to the present invention is to merge the at least onedewarped layer with the other segmented layers back together to form afinal image 790 via a merging algorithm. In this example, the firstlayer of the final image is the background and then all the layer indecreasing order of distance from the camera as calculated by the depthanalysis algorithm 710 are superposed to form the full image 790 withadaptive dewarping. In some embodiments according to the presentinvention, the merging of the at least one dewarped layer with the otherlayers is done by adjusting either the texture mesh or the display mesh.Alternatively, the merging can also be done pixel by pixel. As can beseen in this example figure, this merged final image 790 has some dashedpart 792 on the tree where no information was originally captured by thecamera. The correction of these regions without information will beexplained with FIG. 8 . This missing information is present in thisexample, but in some other examples according to the present invention,if the layers on top were increased in size by the adaptive dewarpingalgorithm, there could be an output images without any missingbackground information as will be explained with FIG. 9 . Also, in someembodiments according to the present invention, at least one of themultiple layers after adaptive dewarping based on image context could befurther processed before merging them together. One example is when theprocessing step further include adding some voluntary blur on at leastone of the depth layers in order to add a bokeh effect depending oncontext and depth instead of the traditional bokeh effect only based ondepth. For example, in the context of an image of a human face in frontof a distant background, this context-based bokeh effect could be addedautomatically to blur the background and keep the human face wellfocused. In other applications of the current invention, when thebackground is more important than the foreground, the opposite couldalso be done, with the background in clear focus and the foregroundobjects blurred for inverted bokeh effect. Also, in some embodimentsaccording to the present invention, the multiple segmented layers afteradaptive dewarping based on context could be further processed beforemerging them together, adding on-purpose some translations, somerotation or some scaling to at least one of the dewarped layers tocreate voluntary perspective or 3D effects. Also, in some otherembodiments according to the present invention, the further processingof the multiple segmented layers before merging them together could alsoinclude a perspective tilt correction of at least one element in asegmented layer in order to correct the perspective with respect eitherto the horizon or any target direction in the scene. This perspectivetilt correction is especially useful when the element is a buildingcaptured with an unpleasantly looking tilt angle in the originalwide-angle image to correct its shape to appears as if it was capturedwithout this tilt angle, but could be applied to any kind of element.Also, in some other embodiments according to the present invention, thefurther processing of the multiple segmented layers before merging themtogether could also include a stabilizing at least one segmented layerin order to avoid unpleasant movement of one or more segmentation layersbetween frames in a video sequence. Also, in some embodiments of themethod of the present invention, some unwanted object layer could beremoved before merging the layers to create the final merged image. Insome embodiment according to the present invention, when more than oneobject are touching each other or close to each other in the originalwide-angle image, like the human 703 and the tree 704 in the example ofFIG. 7 , a specific dewarping process of the segmented layer depends onan adjustable correction weights that can be added to the context-basedadaptive dewarping. These correction weights can adapt the level ofdewarping done on these objects by increasing or decreasing the level ofdewarping to make sure that less important objects does not interferewith more important objects nearby. In the example of FIG. 7 , the tree704 could have a lower correction weight to avoid interfering with thedewarped layer of human 703. These correction weights can be preset inthe device running the adaptive dewarping or be manually adjusted orselected according to a user preference. These correction weights canalso be automatically adjusted by an algorithm, including one trainedvia an artificial intelligence method, the algorithm automaticallyinterpreting the intention of the user based on how the picture iscaptured. This adjustment of the correction weights is especially usefulwhen the user can see a preview of the dewarped image before capturingthe image and he adjust the parameters of the camera accordingly to havea better looking final dewarped image.

FIG. 8 shows an optional method for filling missing image informationduring the adaptive dewarping method based on context and depth layersis applied, sometime called inpainting. This inpainting technique isused to complete at least one part of missing information in theoriginal wide-angle image. In the possible case in which the final imageof the method explained with respect to FIG. 7 has missing information,this further step allows to improve the final image to make it morevisually pleasing for a human observer. The image 800 is an exampleoriginal wide-angle image either from a lens with a rectilinearH=f*tan(e) projection or from a wide-angle lens in which the distortionhas been dewarped to obtain a rectilinear projection. In this exampleimage 800, the setting is indoor with five people in a group selfie (orgroupie) setting. This image 800 is just an example, but this method tofill the missing information is not limited to any setting and could beused with any image on which the adaptive dewarping algorithm is used.The image 800 contains lines 802 from walls that were kept straightbecause the image follows a rectilinear projection. The image 800 alsohas a background wall texture 804 that is partially hidden by people805, 806, 807, 808 and 809. As with the adaptive dewarping method ofFIG. 7 , the first step is the image segmentation and depth analysisstep 810. For simplicity, in this example, the algorithm created twolayers, one layer 820 with all the people standing at relatively thesame distance from the camera and one layer 825 with the background. Thesegmentation and depth layer 820 has no missing information becauseobjects are in the foreground and the segmentation and depth layer 825has missing information because it is in the background. As with theadaptive dewarping method of FIG. 7 , the next step is the context-basedadaptive dewarping 830 to dewarp the warped geometries. The layer 820has people in it and so the adaptive dewarping with body and faceprotection that make people shape visually appealing is used to get thedewarped layer 840. After adaptive dewarping, if the layer were mergeddirectly together as in the method of FIG. 7 , we would obtain the imagewith missing information 860. Compared to the original image 800, theperson 867 in the center was not moved or morphed by the adaptivedewarping process and hence no zone of missing information behind it ispresent. However, the person 866 was moved to the left by the adaptivedewarping process and it created a zone 862 of missing information inthe background. Similarly, the person 868 was moved to the right by theadaptive dewarping process and it created a zone of missing informationin the background. Because they are closer to the camera, the people 865and 869 were enlarged by the adaptive dewarping correcting theperspective compared to their respective image 805 and 809 and hence nozone of missing information was created behind them. Instead of merginglayers 840 and 850, the optional additional step 870 is filling missinginformation with a completion algorithm or inpainting algorithm. Thiscompletion algorithm can use various methods to complete the image,including, but in no way limited to, applying a blur based on thetexture and color surrounding the missing information zone, applying agradient line that gradually change the color from the color on one sideto the color on the other side of the missing information region,usually in the direction that minimize the length of these generatedgradient lines, using an artificial intelligence network trained tocomplete the missing information of pictures or the likes. Thiscompletion algorithm can be executed on a hardware processor consistingof a processing unit (CPU or GPU) located either in the same device asthe camera or in a separate device that receive the output merged image795 according to the method of the present invention. Alternatively, thecompletion algorithm can also be executed in parallel to the adaptivedewarping based on context step 830. The output of the completionalgorithm 870 is the completed layer 885 since layer 850 had missinginformation and the unmodified layer 875 since layer 840 had no missinginformation. The example of FIG. 8 shows one completed layer 885 becauseonly the background had missing information, but there could be manycompleted layers outputted if there were many layers with missinginformation. The layers without missing information 875 are then mergedwith the completed layers 885 in order of depth from the farthest layerto the closest. The result is the final image with filled information890 in which the perspective corrections were applied, the people shapeswere corrected to avoid unpleasant stretching and the missing backgroundinformation was filled with the completion algorithm compared to theimage with missing information 860, obtaining a filled background 892which is visually pleasing. In some embodiment of the present invention,the algorithm can also optionally use the information from any previousframe or even from multiples previous frames from a video sequence tocomplete the missing information of the current frame. This is possiblewhen the missing information of the current frame was visible in anyprevious frame before some movement in the scene created the zone ofmissing information.

FIG. 9 shows an alternate optional method to the method of FIG. 8 forhiding instead of filling missing image information during the adaptivedewarping method based on context and depth layers is applied. With thisexample, at least one part of missing information in the originalwide-angle image is hidden by scaling of the at least one dewarped layerIn the possible case in which the output image of the method explainedwith respect to FIG. 7 has missing information, this alternate methodallows to improve the output image to make it more visually pleasing fora human observer. Starting with the same image 900 as the image 800 fromFIG. 8 , this image is again an example original wide-angle image eitherfrom a lens with a rectilinear H=f*tan(e) projection or from awide-angle lens in which the distortion has been dewarped to obtain arectilinear projection. In this example image 900, the setting is indoorwith five people in a group selfie (or groupie) setting. This image 900is just an example, but this method to hide the missing information isnot limited to any setting and could be used with any image on which theadaptive dewarping algorithm is used. The image 900 contains lines 902from walls that were kept straight because the image follows arectilinear projection. The image 900 also has a background wall texture904 that is partially hidden by people 905, 906, 907, 808 and 909. Aswith the adaptive dewarping method of FIG. 7 , the first step is theimage segmentation and depth analysis step 910. For simplicity, in thisexample, the algorithm created two layers, one layer 920 with all thepeople standing at relatively the same distance from the camera and onelayer 925 with the background. The segmentation and depth layer 920 hasno missing information because objects are in the foreground and thesegmentation and depth layer 925 has missing information because it isin the background. As with the adaptive dewarping method of FIG. 7 , thenext step is the context-based adaptive dewarping 930. The layer 920 haspeople in it and so the adaptive dewarping with body and face protectionthat make people shape visually appealing is used to get the dewarpedlayer 940. After adaptive dewarping, if the layer were merged directlytogether as in the method of FIG. 7 , we would obtain the image withmissing information 960. Compared to the original image 900, the person967 in the center was not moved or morphed by the adaptive dewarpingprocess and hence no zone of missing information behind it is present.However, the person 966 was moved to the left by the adaptive dewarpingprocess and it created a zone 962 of missing information in thebackground. Similarly, the person 968 was moved to the right by theadaptive dewarping process and it created a zone of missing informationin the background. Because they are closer to the camera, the people 965and 969 were enlarged by the adaptive dewarping correcting theperspective compared to their respective image 905 and 909 and hence nozone of missing information was created behind them. Instead of merginglayers 940 and 950 or filling the missing information as in the methodof FIG. 8 , the optional additional step 970 is hiding zones of missinginformation with an algorithm to adjust relative magnification. Thismethod adjusts the relative magnification of objects in the front layersand enlarges them just enough so that no zone of missing informationwill be present in the background when combining the layers togethers.This algorithm to adjust the relative magnification on some layers canbe executed on a hardware processor consisting of a processing unit (CPUor GPU) located either in the same device as the camera or in a separatedevice that receive the output merged image 795 according to the methodof the present invention. Alternatively, the hiding of missinginformation by the algorithm adjusting relative magnification can alsobe executed in parallel to the adaptive dewarping based on context step930. The output of the algorithm to adjust relative magnification 970 isthe magnified layer 975 since layer 940 had objects that were moved,creating zones of missing information in the merged image 960 and theunmodified layer 985 since layer 950 was in the background and no changeof magnification was required. In this example, the person in the center943 was not adjusted by the adaptive dewarping algorithm 930 and hencecreated no zone of missing information. For this reason, the person 978remain unchanged in the layer 975. For the people 942 and 944 that wererespectively moved to the left and to the right by the adaptivedewarping algorithm 930, they need to be magnified by the algorithm 970to hide the zones of missing information created when they were movedand hence the people 977 and 979 in the resulting layer are enlarged.Also, in this example, the people 941 and 945 were also enlarged intorespectively people 976 and 980 even if there was no zone of missinginformation behind them. This specific case is to show that thealgorithm to adjust relative magnification 970 can even adjust objectsor layers without missing information close to them in order to keep theoverall proportions of the image respected. The example of FIG. 9 showsone layer with adjusted magnification 975, but there could be manylayers with adjusted magnification outputted if there were many layerswith objects needing enlargement to hide zones of missing information.The layers with adjusted magnification 975 are then merged with thebackground layers 985 in order of depth from the farthest layer to theclosest. The result is the image with some foreground objects enlarged990 in which the perspective corrections were applied, the people shapeswere corrected to avoid unpleasant stretching and the missing backgroundinformation was hidden by the algorithm to adjust relative magnificationcompared to the image with missing information 960. In the resultingimage 990, there is no missing background around the area 992 comparedto zone of missing information 962, which is visually pleasing.

In some other embodiment according to the present invention, both thecompletion method of FIG. 8 and the hiding method by adjusting relativemagnification of FIG. 9 can be used together to minimize their relativeimpact on the image.

FIG. 10 shows the method according to some embodiments of the currentinvention in which the dewarping projection of the background layer candepend on the detected context of the objects in the foreground. In thisexample, the processing of a segmented layer uses a dewarping projectionfor a background layer depending on a detected context of the originalwide-angle image The figure shows two example images 1000 and 1050, asthey would appear when captured by a lens having a resolution curve likethe one of FIG. 3 . Because of the distortion profile, the backgroundslines 1010 and 1060 have visible curved lines for both the horizontaland vertical lines as explained previously at FIG. 4 . For the specificexample of image 1000, since there are humans 1015, 1016, 1017, 1018 and1019 inside the picture, the algorithm 1020 for context-based backgrounddewarping would detect that the humans are in a group selfie, orgroupie, scenario and that the ideal background dewarping would be acylindrical projection. The output of the dewarping algorithm 1020 isthe background layer 1030 in which the background lines 1040 shows thatvertical lines in the object are straight in the image, but horizontallines in the object are curved in the image, as in a cylindricalprojection. In the case when the background layer has some missinginformation because it was hidden by a foreground object, an optionalinpainting technique can be used to complete the image if needed for thefinal output, as represented by the dashed lines 1045. Next, in theexample image 1050, the same background 1060 as the previous background1010 is visible, but this time without any human in front in theforeground. In this case, the algorithm 1070 for context-basedbackground dewarping would detect that because it is an indoor scene,keeping straight lines in the scene as straight lines in the image ispreferred and the output projection should be rectilinear. The output ofthe dewarping algorithm 1070 is the background layer 1080 in which thebackground lines 1090 are straight in the image as in a rectilinearprojection. The cylindrical and rectilinear projections ideal output inthis figure are only examples of background projections that could beideal for a given context, but the method according to the presentinvention is not limited to any specific projection and could be any ofstereographic, equidistant, equisolid, orthographic, Mercator or anyother custom projection.

FIG. 11 shows one example implementation of the algorithm according tothe method for context-based adaptive dewarping based on depth andsegmentation layer for ideal face protection. In this exampleimplementation of the algorithm, the processing of the at least onesegmented layer to create the at least one dewarped layer includescreating a virtual camera centered on the element having the warpedgeometry, applying a rectilinear correction on the virtual camera andtranslating the result to a correct location in the final image. Theexample algorithm starts with an original wide-angle image 1100. Thisoriginal wide-angle image can be of any field of view, but generally thewarped geometries shown in FIG. 2 and FIG. 4 are mostly visible inwide-angle images with over 60° of full field of view. In an embodimentaccording to the present invention, this original wide-angle image 1100is directly captured with a camera module having a wide-angle lens withor without departure from the rectilinear projection. In some otherembodiments according to the present invention, the original image 1100has already been processed by a processor to either correct the originaldistortion from the camera module, improve the image quality or applyingany other image processing. Alternatively, the original wide-angle image1100 could be combined inside a processor from multiple narrow-angleimages or completely computer-generated. In this example, in no waylimiting the scope of the invention, the original wide-angle image has abackground 1110 consisting of a mountain landscape and a human face 1115in the foreground as the object. Because the human face is close to thecorner, the face is stretched, and the original object proportions arenot kept. The original wide-angle image 1100 is not visually pleasing asalready explained in FIG. 4 . The context-based adaptive dewarpingmethod does a segmentation and classification of the human face 1115.The next step 1120 in this example algorithm is creating a virtualcamera 1130 with the human face 1135 centered in it. This virtual camera1130 has a narrow field of view and is rotated as if it was in thecenter of the original image, fixing the stretching because in a narrowfield of view in the center of the image, the original proportion arekept. This is represented by the circular head shape, representing theideal proportion in this example. Mathematically, this example step 1120of rotating the virtual camera, is described as follow. Each point Pinin the segmented layer of the original wide-angle image is assigned acoordinate (x,y) in the original wide-angle image.

${Pin} = \begin{pmatrix}x \\y\end{pmatrix}$

The center position of the face in the original wide-angle image isPin₀, having coordinate (x₀, y₀).

${Pin}_{0} = \begin{pmatrix}x_{0} \\y_{0}\end{pmatrix}$

Euler angles (α,θ) are calculated from the center position Pin₀according to optical distortion using a function F.

(α,θ)=F(Pin ₀)

For each input point Pin, a virtual camera projection position in the 3Dspace Pin_(3d) having coordinates (x′,y′,z′) is calculated with aconversion function called P_(camera).

${Pin}_{3d} = \begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime}\end{pmatrix}$Pin _(3d) =P _(camera)(Pin)

Next, the rotation of the virtual camera is done in this examplealgorithm by multiplying a rotation matrix M which invert the Eulerangles on each input point Pin₃d in 3D space to obtain a positionPout_(3d) having coordinates (u′,v′,w′).

${{Pout}_{3d} = \begin{pmatrix}u^{\prime} \\v^{\prime} \\w^{\prime}\end{pmatrix}}{{Pout}_{3d} = {{M\left( {{- \alpha},{- \theta}} \right)}*{Pin}_{3d}}}$

This position Pout_(3d) in 3D space is then converted to a position in2D space Pout having coordinate (u,v) using an inverse functionP⁻¹display.

${{Pout} = \begin{pmatrix}u \\v\end{pmatrix}}{{Pout} = {P_{display}^{- 1}\left( {Pout}_{3d} \right)}}$

The next step 1140 of this example algorithm is to translate the resultfrom the virtual camera 1130 back to the original position in the imagebefore the rotation of the virtual camera, giving the frame 1150 inwhich the human face 1155 has ideal proportions, but may still berotated or of the wrong size for perfect match with the background.Mathematically, we use a translation vector T=(t_(x),t_(y)) to calculatethe position Pout′ for each point of the segmented layer.

Pout′=Pout+T

The next step 1160 of this example algorithm is optional and it includesany further processing by the adaptive dewarping algorithm of thevirtual camera 1170 in order to improve the final projection of thehuman face 1175, including rotation, scaling or any other transformationfor ideal face protection. Mathematically, we apply an optional rotationmatrix R and a scale matrix S to Pout′ in order to calculate a finalposition Pout” for each point of the segmented layer.

Pout″=R*S*Pout′

The last step 1180 of this example algorithm is merging back thesegmented human face layer 1195 to the other layers, represented here bythe background layer with the mountain range 1190. When the layers aremerged back together, the texture mesh or the display mesh can beadjusted as desired for the best fit between the merged layers. Thismethod for face protection was shown as an example, but the method canbe applied for object protection of any kind according the presentinvention. The algorithm described in this FIG. 11 is only an exampleimplementation and is not limiting. Other algorithms could be used toachieve the same result while keeping with the spirit of the invention.

FIG. 12 shows an example embodiment of a physical device 1230 capturingthe original wide-angle image, processing it according to the method ofthe present invention to enhance the image based on image context andoutputting the final image on a display screen. An object scene 1200 isvisible to the physical device 1230, meaning that some rays of lightfrom the object scene, here shown by the two extreme rays 1210 and 1212defining the field of view of the imager, are reaching the imager 1220of the physical device 1230. In this example, the imager 1220 is awide-angle lens having a field of view generally larger than 60° andforming an optical image in an image plane with an image sensor locatedgenerally at the image plane of the wide-angle lens and transforming therays of light from the optical image to a digital image filerepresenting the original wide-angle image 1240. This originalwide-angle image file has a plurality of elements in either theforeground or the background, with at least one element having warpedgeometries, visible for example by the stretched face of the person1245. In other embodiments, the imager could consist of any other waysof creating a digital image, including other optical system with lens,mirrors, diffractive elements, meta-surfaces or the likes or anyprocessor creating or generating a digital image file from any source.This embodiment is only an example of such a physical device accordingto the present invention and this example does not limit the scope ofthe present invention. This physical device could be any devicecomprising a way to receive an original wide-angle image 1240, processit and display it, like a smartphone, a tablet, a laptop or desktoppersonal computer, a portable camera, or the likes. In this example, thephysical device 1230 also comprise a processor 1250 able to executealgorithms to process the original wide-angle image 1240 to a finalimage, including segmentation, classification, dewarping at leastpartially, merging the layers, other various image quality processingand enhancement or the likes. In this example, the processor 1250 is acentral processing unit (CPU), but in other embodiments, the processingcould be done by any kind of processor, including a CPU, a GPU, a TPU,an ASIC, a FPGA or any other hardware processor configured to executesoftware algorithm for performing the stated functions or able toprocess digital image files. The processor 1250 then output the finalimage 1270 to a display 1260. The final image 1270 has dewarpedgeometries, visible for example by the correct proportions of the faceof human 1275. In this example, the display 1260 is part of the physicaldevice 1230, like the screen of a smartphone or the likes, but in otherembodiment, the final image file alternatively could be transferred toany other device for either display or analysis by another algorithm.This example of a single physical device 1230 comprising the imager1220, the processor 1250 and the display 1260 is just an exampleembodiment according to the present invention, but these three featurescould also be part of multiple physical devices with the digital imagefile exchanged between them via any communication link to share digitalimage file, including, but not limited to, a computer main bus, a harddrive, a solid-state drive, a USB drive, transferred over the air viaWi-Fi or any other way of transferring digital image file betweenmultiple physical devices.

In some embodiments according to the present invention, the adaptivedewarping method based on segmentation and depth layers is used toeither maximize the straightness of the lines in the images compared tothe original lines in the object scene, maximize the output image fullfield of view compared to the original image full field of view and/ormaximize the conservation of the proportions in the output imagecompared to the real proportions in the object scene. When the method isused to maximize the straightness of the lines in the images compared tothe original image file, the context-based adaptive dewarping method 760transform the various segmentation and depth layers with a priority onmaking the straight lines in the object scene as straight as possible inthe merged image 790. When the method is used to maximize the outputimage full field of view compared to the original image full field ofview, a special dewarping is targeted for the corners of the originalimage in order to make sure the output merged image diagonal full fieldof view is kept as close as possible to the original image diagonal fullfield of view. This is done in order to avoid losing information in theimage by reducing the field of view forcing to crop the corners or avoidcreating a black corner with no information or black sides of the outputimage with no information. In order to keep the field of view as closeas possible between the original image and the output image, the specialdewarping in the corners can ignore the segmentation or the depth layersand not apply a specific dewarping based on context in that zone in thecorners of the image. The choice of not applying the specific adaptivedewarping based on context and depth layer in that zone or in anotherzone of the image or on a specific layer for any other reason is alsopossible according to the present invention. When the method is used tomaximize the conservation of the proportions in the output imagecompared to the real proportions in the object scene, the context-basedadaptive dewarping method 760 transform the various segmentation anddepth layers with a priority on the proportion. In this case, theproportions in the output merged image 790 all appears similar to theproportions in the real object scene as would be required when human arevisible in the image. In some embodiment according to the presentinvention, all three of these cases are maximized together.

In all embodiments according to the present invention, the adaptivedewarping algorithm can optionally use information from any previousframe in a video sequence to do temporal filtering. With temporalfiltering, the final output from the adaptive dewarping can be smootherby removing potential artefacts that could be created from a badinterpretation of the algorithm on a specific frame by favoring thetemporal consistency instead of results with large departures from theprevious frames. Temporal filtering is also useful in cases where someshaking of the camera or of part of the object scene would otherwisecreate artefacts.

All of the above are figures and examples show the adaptive dewarpingmethod. In all these examples, the imager, camera or lens can have anyfield of view, from very narrow to extremely wide-angle. These examplesare not intended to be an exhaustive list or to limit the scope andspirit of the present invention. It will be appreciated by those skilledin the art that changes could be made to the embodiments described abovewithout departing from the broad inventive concept thereof. It isunderstood, therefore, that this invention is not limited to theparticular embodiments disclosed, but it is intended to covermodifications within the spirit and scope of the present invention asdefined by the appended claims.

We claim:
 1. A method for enhancing an image based on image context andsegmentation layers, the method comprising: a. receiving, by aprocessor, an original image created by an imager and having a pluralityof elements, each element being in one of a foreground or a backgroundof the original image; b. segmenting, by the processor, the originalimage into a plurality of segmented layers, each of the segmented layersincluding at least one of the elements, the segmentation being based onat least one of a shape of one or more of the elements, a position ofone or more of the elements in the original image, or a depth of one ormore of the elements compared to other elements; c. processing, by theprocessor, at least one of the segmented layers, creating at least oneprocessed layer; and d. merging, by the processor, the at least oneprocessed layer with the other segmented layers to form a final image.2. The method of claim 1, wherein the original image is captured by theimager, the imager being an optical system comprising at least a cameraand a lens.
 3. The method of claim 2 wherein the diagonal field of viewof the lens is at least over 60°.
 4. The method of claim 1, wherein atleast one of the elements is in the foreground of the original image,and the segmentation of the at least one foreground element is based onthe relative depth compared to other elements, the relative depth beingcalculated by an artificial intelligence neural network.
 5. The methodof claim 1, wherein the processing step further includes at least one ofblurring the at least one processed layer, rotating the at least oneprocessed layer, translating the at least one processed layer, scalingthe at least one processed layer, correcting the perspective tilt of theat least one processed layer or stabilizing the at least one processedlayer.
 6. The method of claim 1, wherein an inpainting technique is usedto complete at least one part of missing information in the originalimage.
 7. The method of claim 1, wherein at least one part of missinginformation in the original image is hidden by scaling of the at leastone processed layer.
 8. The method of claim 1, wherein merging the atleast one processed layer with the other layers is done by adjustingeither a texture or a display mesh.
 9. A device for enhancing an imagebased on image context and segmentation layers, the device comprising:a. an imager creating an original image having a plurality of elements,each element being in one of a foreground or a background of theoriginal image; and b. a processor configured to: i. segment theoriginal image into a plurality of segmented layers, each of thesegmented layers including at least one of the elements, thesegmentation being based on at least one of a shape of one or more ofthe elements, a position of one or more of the elements in the originalimage, or a depth of one or more of the elements compared to otherelements, ii. process at least one of the segmented layers, creating atleast one processed layer, and iii. merge the at least one processedlayer with the other segmented layers to form a final image.
 10. Thedevice of claim 9, wherein the imager is an optical system comprising atleast a camera and a lens.
 11. The device of claim 10 wherein thediagonal field of view of the lens is at least over 60°.
 12. The deviceof claim 9, wherein at least one of the elements is in the foreground ofthe original image, and the segmentation of the at least one foregroundelement is based on the relative depth compared to other elements, therelative depth being configured to be calculated by an artificialintelligence neural network.
 13. The device of claim 9, wherein theprocessor, in the processing step, is further configured to at least oneof blur the at least one processed layer, rotate the at least oneprocessed layer, translate the at least one processed layer, scale theat least one processed layer, correct the perspective tilt of the atleast one processed layer or stabilize the at least one processed layer.14. The device of claim 9, wherein the processor is configured to use aninpainting technique to complete at least one part of missinginformation in the original image.
 15. The device of claim 9, whereinthe processor is configured to hide at least one part of missinginformation in the original image by scaling of the at least oneprocessed layer.
 16. The device of claim 9, wherein the processor isconfigured to merge the at least one processed layer with the otherlayers by adjusting either a texture or a display mesh.